Parallel computing is the distribution of a computing operation among a plurality of processors and/or a plurality of computing systems. Parallel computing is advantageous because a computationally expensive processing task may take less time to complete if more processors are used. For example, scientific and technological research frequently comprises computational tasks which, due to their complexity, require years to complete on a single processor. However, such tasks can frequently be completed in a manageable amount of time if divided among a large number of processors. Weather forecasting and computerized simulations of real-world phenomena also frequently comprise complex computational tasks which may benefit from parallel computing. Parallel computing is particularly advantageous for high performance computing, a term used in the art to denote computing tasks requiring very large amounts of computational resources.
In the message passing model, a computing task is divided among a plurality of processes in predefined ways. The participating processes communicate with each other by sending and receiving messages. For parallel computing to succeed, it is important for this inter-process communication to be effective. Thus, the processes should ideally communicate with each other according to predefined communication protocols.
The Message Passing Interface, or MPI, is a protocol standard known in the art for facilitating communication between a plurality of processors cooperating on a computing task. MPI is widely used to implement parallel computing systems and applications based on the message passing model. MPI defines semantics of various types of communications. Specifically, MPI defines sending and receiving interfaces for point-to-point communications between two processes. MPI also defines a set of operations for collective communications among multiple processes. MPI is the de facto standard for communication interfaces in parallel computing.